Methods & Results for Behavioural Study of Moral Judgements of Humans vs. AI

Methods

Participants

Number of participants at start: 93

Number of participants after cleaning: 90

Participants removed during data cleaning timed out.

N = 90

Mage = 46.63 , SDage = 14.75

Age range = 20 – 81 years

Gender Composition of Sample

Gender n %
Man 38 42.2
Non-binary 2 2.2
Prefer Not To Say 1 1.1
Woman 49 54.4

Ethnic / Cultural Identities

Because the ethnicity question used UK-specific categories but the sample was US-based, free-text responses were cleaned and recoded into broader categories based on standard US Census categories, with an additional category for those who identified with multiple ethnicities/cultural identities. The table below summarises the cleaned and recoded responses.

Appendix A: Table Summarising Participants’ Cultural Identities

US Census Ethnic/Cultural Categories n %
White 58 58 (64.4%)
Black / African American 10 10 (11.1%)
Multiple Ethnic/Cultural Identities 9 9 (10%)
Asian 5 5 (5.6%)
Other 5 5 (5.6%)
Southwest Asian / North African 3 3 (3.3%)

Nationalities

For this study, the sample was fixed to US nationals & residents. Here I run a quick check to ensure that this was the case.

Number of participants who identified as US nationals:

90 out of 90 participants

Number of participants who stated their country of residence is the United States:

90 out of 90 participants

Nationality Verification
• US nationals: 90 / 90 (Total Sample)
• US residents: 90 / 90 (Total Sample)

Results

Main Analysis (All Data)

First, the GLMM model was fit with a full random effects structure.

full.hVai <- glmer(MJ ~ PBH*PDE*Agent +  (1+PBH*PDE*Agent|ID) + (1+Agent|dilemma),  data=df, family = binomial(link = "logit"),
               control = glmerControl(optCtrl = list(maxfun = 2e5), optimizer = "bobyqa"))

This returned a singularity issue. Therefore, the complexity of the random effects structure was reduced (by order of importance) until no singularity warning was returned. The model below was the final fitted model.

hVai <- glmer(MJ ~ PBH*PDE*Agent +  (1+PBH+PDE|ID) + (1|Dilemma),  data=df, family = binomial(link = "logit"),
              control = glmerControl(optCtrl = list(maxfun = 2e5), optimizer = "bobyqa"))

#OR coefficients 
OR_hVai <- exp(fixef(hVai))
#CIs
ci_hVai <- confint(hVai, method = "Wald")
ci_hVai.or <- exp(ci_hVai)
ses_hVai <- sqrt(diag(vcov(hVai)))


Table 1. Logistic GLMM Human v AI MJs explained by MG (OR)


Table 2. Logistic GLMM Human v AI MJs explained by MG (Log Odds)


Main Effects Plots


Interaction Effects Plots


Simple Effects Analysis of Main Model (by factor)

Estimated Marginal Means

Table 5. Estimated marginal means of PDE within PBH

Table 3. Estimated marginal means by PBH × PDE within Agent


Table 4. Estimated marginal means by PDE x Agent within PBH


Table 5. Estimated marginal means by Agent × PDE within PDE



Contrasts of EMMs

Table 8. Pairwise contrasts for PDE within PBH

[1] “

Table 6. Pairwise contrasts for PBH × PDE within Agent


Table 7. Pairwise contrasts for PDE × Agent within PBH


Table 8. Pairwise contrasts for PBH × Agent within PDE


Two Sided Equivalence Test for Main Analysis


Table 9. TOST-test for Practical Equivalence for Main Model



Figure 9.
TOST results for Main Analysis

Note. Bounds set to mid-point value of small effect size boundary for log odds (±0.94). This is what the study was actually powered to detect for a three-way interaction.


Model Comparisons


Table 10. Comparison of logistic GLMMs with alternative fixed-factor combinations predicting moral judgments (with RE structure as in final model)

Note. Lower AIC and BIC indicate better fit.


Model Checks



Removing dilemmas with ambiguous MG predictions

The additional dilemmas developed in-lab did not have explicit predictions in the MG model. For the most part, these could be estimated based off of the predictions made for alternative dilemmas, and/or guided by the fulfilments of PBH/PDE. However, the unprohibited better alternative equipment-only disproportionate cost intended and the unprohibited better alternative disproportionate cost scenarios had particularly ambiguous estimates. Therefore, for the first study and this study, I conduct analyses with, and without, these dilemmas. Until, at least there is sufficient information available to provide more appropriate predictions.

First, the GLMM model was fit with a full random effects structure.

full.hVai <- glmer(MJ ~ PBH*PDE*Agent +  (1+PBH*PDE*Agent|ID) + (1+Agent|dilemma),  data=reduced_df, family = binomial(link = "logit"),
                   control = glmerControl(optCtrl = list(maxfun = 2e5), optimizer = "bobyqa"))

This returned a singularity issue. Therefore, the complexity of the random effects structure was reduced (by order of importance) until no singularity warning was returned. The model below was the final fitted model.

hVai <- glmer(MJ ~ PBH*PDE*Agent +  (1+PBH+PDE+Agent|ID) + (1|Dilemma),  data=reduced_df, family = binomial(link = "logit"),
              control = glmerControl(optCtrl = list(maxfun = 2e5), optimizer = "bobyqa"))

This model returned no singularity error but the correlations between the RE slopes and the intercept were very high (0.82-0.95), suggesting overfitting. Therefore, the random effects structure was further reduced by removing the random slope for Agent. This returned no singularity warning and much lower correlations between the RE slopes and intercept.

hVai <- glmer(MJ ~ PBH*PDE*Agent +  (1+PBH+PDE|ID) + (1|Dilemma),  data=reduced_df, family = binomial(link = "logit"),
              control = glmerControl(optCtrl = list(maxfun = 2e5), optimizer = "bobyqa"))

#OR coefficients 
OR_hVai <- exp(fixef(hVai))
#CIs
ci_hVai <- confint(hVai, method = "Wald")
ci_hVai.or <- exp(ci_hVai)
ses_hVai <- sqrt(diag(vcov(hVai)))

Table 2. Logistic GLMM Human v AI MJs explained by MG (OR)

Table 1. Logistic GLMM Human v AI MJs explained by MG (Log Odds)

Main Effects Plots


Interaction Effects Plots


Simple Effects Analysis of Main Model (by factor)

Estimated Marginal Means

Table 5. Estimated marginal means by PDE within PBH


Table 3. Estimated marginal means by PBH × PDE within Agent


Table 4. Estimated marginal means by PDE x Agent within PBH


Table 5. Estimated marginal means by Agent × PBH within PDE



Contrasts of EMMs

Table 8. Pairwise contrasts for PDE within PBH


Table 6. Pairwise contrasts for PBH × PDE within Agent


Table 7. Pairwise contrasts for PDE × Agent within PBH


Table 8. Pairwise contrasts for PBH × Agent within PDE


Two Sided Equivalence Test for Main Analysis (Reduced Dilemmas)


Table 9. TOST-test for Practical Equivalence for Model with Reduced Dilemmas



Figure 9.
TOST results for Analysis of Model with Reduced Dilemmas

Note. Bounds set to mid-point value of small effect size boundary for log odds (±0.94). This is what the study was actually powered to detect for a three-way interaction.


Model Comparisons


Table 10. Comparison of logistic GLMMs with alternative fixed-factor combinations predicting moral judgments (with RE structure as in final model)

Note. Lower AIC and BIC indicate better fit.


Model Checks



Removing Participants who failed a costless rescue Human scenario

First, the GLMM model was fit with a match random effects structure.

full.hVai <- glmer(MJ ~ PBH*PDE*Agent +  (1+PBH*PDE*Agent|ID) + (1+Agent|dilemma),  data=df_match, family = binomial(link = "logit"),
                      control = glmerControl(optCtrl = list(maxfun = 2e5), optimizer = "bobyqa"))

This returned a singularity issue. Therefore, the complexity of the random effects structure was match (by order of importance) until no singularity warning was returned. The model below was the final fitted model.

hVai <- glmer(MJ ~ PBH*PDE*Agent +  (1+PBH+PDE|ID) + (1|Dilemma),  data=df_match, family = binomial(link = "logit"),
              control = glmerControl(optCtrl = list(maxfun = 2e5), optimizer = "bobyqa"))

#OR coefficients 
OR_hVai <- exp(fixef(hVai))
#CIs
ci_hVai <- confint(hVai, method = "Wald")
ci_hVai.or <- exp(ci_hVai)
ses_hVai <- sqrt(diag(vcov(hVai)))

Table 2. Logistic GLMM Human v AI MJs explained by MG (OR)

Table 1. Logistic GLMM Human v AI MJs explained by MG (Log Odds)

Main Effects Plots


Interaction Effects Plots


Simple Effects Analysis of Main Model (by factor)

Estimated Marginal Means

Table 5. Estimated marginal means by PDE within PBH


Table 3. Estimated marginal means by PBH × PDE within Agent


Table 4. Estimated marginal means by PDE x Agent within PBH


Table 5. Estimated marginal means by Agent × PBH within PDE



Contrasts of EMMs

Table 7. Pairwise contrasts for PDE within PBH


Table 6. Pairwise contrasts for PBH × PDE within Agent


Table 7. Pairwise contrasts for PDE × Agent within PBH


Table 8. Pairwise contrasts for PBH × Agent within PDE


Two Sided Equivalence Test for Main Analysis


Table 9. TOST-test for Practical Equivalence for Main Model



Figure 9.
TOST results for Main Analysis

Note. Bounds set to mid-point value of small effect size boundary for log odds (±0.94). This is what the study was actually powered to detect for a three-way interaction.


Model Comparisons


Table 10. Comparison of logistic GLMMs with alternative fixed-factor combinations predicting moral judgments (with RE structure as in final model)

Note. Lower AIC and BIC indicate better fit.


Model Checks



Removing Participants who failed CR Human scenario and dilemmas with ambiguous MG predictions.

First, the GLMM model was fit with a reduced_match random effects structure.

full.hVai <- glmer(MJ ~ PBH*PDE*Agent +  (1+PBH*PDE*Agent|ID) + (1+Agent|dilemma),  data=reduced_df_match, family = binomial(link = "logit"),
                      control = glmerControl(optCtrl = list(maxfun = 2e5), optimizer = "bobyqa"))

This returned a singularity issue. Therefore, the complexity of the random effects structure was reduced_match (by order of importance) until no singularity warning was returned. The model below was the final fitted model.

hVai <- glmer(MJ ~ PBH*PDE*Agent +  (1+PBH+PDE|ID) + (1|Dilemma),  data=reduced_df_match, family = binomial(link = "logit"),
              control = glmerControl(optCtrl = list(maxfun = 2e5), optimizer = "bobyqa"))

#OR coefficients 
OR_hVai <- exp(fixef(hVai))
#CIs
ci_hVai <- confint(hVai, method = "Wald")
ci_hVai.or <- exp(ci_hVai)
ses_hVai <- sqrt(diag(vcov(hVai)))

Table 2. Logistic GLMM Human v AI MJs explained by MG (OR)

Table 1. Logistic GLMM Human v AI MJs explained by MG (Log Odds)

Main Effects Plots


Interaction Effects Plots


Simple Effects Analysis of Main Model (by factor)

Estimated Marginal Means

Table 5. Estimated marginal means by PDE within PBH



Table 3. Estimated marginal means by PBH × PDE within Agent


Table 4. Estimated marginal means by PDE x Agent within PBH


Table 5. Estimated marginal means by PBH within PDE



Contrasts of EMMs

Table 7. Pairwise contrasts for PDE within PBH


Table 6. Pairwise contrasts for PBH × PDE within Agent


Table 7. Pairwise contrasts for PDE × Agent within PBH


Table 8. Pairwise contrasts for PBH × Agent within PDE


Two Sided Equivalence Test for Main Analysis


Table 9. TOST-test for Practical Equivalence for Main Model



Figure 9.
TOST results for Main Analysis

Note. Bounds set to mid-point value of small effect size boundary for log odds (±0.94). This is what the study was actually powered to detect for a three-way interaction.


Model Comparisons


Table 10. Comparison of logistic GLMMs with alternative fixed-factor combinations predicting moral judgments (with RE structure as in final model)

Note. Lower AIC and BIC indicate better fit.


Model Checks